Qualitative Study of GPR Model Performance using Increasing Dataset Sizes

Alessio Tamburro


The source code and the full notebook (including all functions) for this article are available here.

Note: the html version of this notebook was created with nbconvert and running the command jupyter nbconvert --to html --TagRemovePreprocessor.remove_input_tags="notvisible" notebook.ipynb. A tag "notvisible" was added to the cells that are not displayed in this rendered html

This study follows the post and notebook published earlier here. In that post, we used Optuna to search the space of potential polynomial kernels with varying complexity and find an optimal kernel (automated kernel engineering) for the Gaussian Process model. In this post, we will use the same approach to train Gaussian Process regression models for a set of 24 benchmark functions, which are used to generate the training data. We will observe how increasing dataset sizes affects the performance of the trained models. The study focuses on 1D functions. The plan is to extend this study to more input dimensions.

Summary of the Results

Before digging deeper into the details, we show the summary of this study below.

Data Generation using Benchmark Functions

In this section, we show how we generate data based on an example benchmark function, the dropwave function. This function along with several other functions are typically used to benchmark optimization algorithms. We utilized some of the benchmark functions that are available from SFU. These functions will be considered our true generative process function.

The function generate_data generates a grid of input values to sample from the dropwave function, whhich is defined as a lambda function. The grid is built using the ranges for the input variable X1, which are specified in the design space dictionary. The design space can also specify if we want to generate the data with noise. We will come back to this point later in this notebook. For now, we set the noise ('err') to 0.

The selected training dataset has 10 data rows.

The full generated dataset has 1000 data rows.

Gaussian Process Regression Model Training

We can now fit a Gaussian Process regression model to the training data generated above using the Python library of scikit-learn for Gaussian Process regression gaussian_process. We let the optimizer find the best paramter values for the Gaussian Process regression model kernel. The optimizer will adapt the parameters to maximize the log marginal likelihood. We selected the RBF kernel for this exercise.

As we can see from the model performance indicators and the graph below, 10 data points do not return a satisfactory model. Later, we will show how the performce changes by iteratively increasing the dataset size.

Adding more points

Given the training array 10 points, we generate 5 additional points on the input grid that satisy the maxmin property. This approach allows for sampling the generating true function where we lack data. The function _generatesamples generates many random data points on the specified grid before applying the minmax condition to select the additional data points.

As we can see, adding one point improves the model performance. We can also notice that the point is added between -1 and 0, which was the region that was not sampled before.

Sequential Training of 1D Benchmark Functions

To assess qualitatively the performance of trained Gaussian Process regression models using increasing dataset sizes, we set up sequential training based on different benchmark functions and increasing dataset sizes. For different benchmark functions, the underlying polynomial kernel for the GPR model will be selected using automated kernel engineering as described in a previous post here (the notebook is available here under the notebook folder).

1D Benchmark Functions

We prepared a dataset that contains also the associated lambda functions we need to generate our training observations.

Sequential Training

Sequential training starts with 10 initial generated data points and proceeds until it reaches 40 data points. If the performance of the model (measured with the R squared) stays above 0.98 for more than 5 subsequent added points, the iterative loop stops. For the polynomial kernel, we use the kernel functions that are available in sklearn: RBF, RationalQuadratic, ExpSineSquared, Matern, DotProduct. The polynomial kernel will have no more than 3 additive terms and each additive term will have no more than 2 product terms. The output of this cell is omitted.

Some Words about Uncertainty on X and Y

We end this notebook by showing how the uncertainty of the measurements of the input and target values can affect the performance of the GPR model. Depending on the magnitude of the uncertainty, achieving reasonable predictive performance for the trained model may require more iterations and/or measurements during sequential training. In fact, this is the situation that most commonly happens in scientific experimentation.

In the example, we add an uncertainty of 0.05 to both the input X1 and the output Z when generating the data. As we can see, even with 30 points the model shows a large uncertainty.

Here, we considered 50 measurements and added a noise kernel term to fit our model.

Copyright (c) [2024] [Alessio Tamburro]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

MIT license